International Review of Research in Open and Distributed Learning 
Volume 16, Number 5 


September - 2015 


Evaluation Criteria for Interactive E-Books for Open 
and Distance Learning 


Aras Bozkurt and Mujgan Bozkaya 

Anadolu University, Turkey 


Abstract 

The aim of this mixed method study is to identity evaluation criteria for interactive e-books. To 
find answers for the research questions of the study, both quantitative and qualitative data were 
collected through a four-round Delphi study with a panel consisting of 30 experts. After that, a 
total of 20 interactive e-books were examined with heuristic inquiry methodology. In the final 
phase, the results of the Delphi technique and the heuristic inquiry results were integrated. As a 
result, four themes, 15 dimensions, and 37 criteria were developed for interactive e-books. Lastly, 
the results and their implications are discussed in this paper and suggestions for further research 
are presented. 

Keywords: Interactive e-books, evaluation criteria, Delphi study, heuristic inquiry, open and 
distance learning (ODL) 


Introduction 

Open and Distance Learning (ODL) strives to provide effective, efficient, engaging, and enduring 
learning opportunities which are dependent on improvements and developments in information 
and communication technologies (ICT). There have been attempts to employ ICT to eliminate the 
limitations that derive from physical and psychological distance among learners, learning sources, 
and learning environments. The influence of ICT resulted in online and digital solutions that 
increase interaction in the learning process. Beldarrain (2006) suggests that technology has 
played a critical role in changing the dynamics of each delivery option over the years, as well as 
the pedagogy in ODL. As new technologies emerged, instructional designers and educators had 
unique opportunities to foster interaction and collaboration among learners, thus creating a true 
learning community. 
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As a result of these developments, e-books and, following that, interactive e-books have gained a 
wide interest and have been used as a valuable and viable medium in both traditional education 
and ODL. By realizing the potential of digital books, institutions of higher education have begun 
to provide interactive e-books for learners to be able to deliver the information in a more effective 
and attractive way. According to Rothman (2006), for distance educators as well as traditional 
classroom educators, digital books would not only enhance student access to information, but 
would also help revolutionize the processes of reading, analyzing, and researching. 

As a response to current developments in the e-book and interactive e-book landscape, interactive 
e-books are defined and their pros and cons are explained in this study. Following that, 
interactivity in interactive e-books is discussed and finally evaluation criteria of interactive e- 
books are explained based on a mixed method study in which Delphi technique and heuristic 
inquiry were used. 


Aim of the Study 

Interactive e-books are used for providing flexibility and presenting enriched content by means of 
hard and soft technologies. As an emerging technology, it is a necessity to define evaluation 
criteria of interactive e-books and contribute to relevant literature. 

On this basis, the main purpose of this research is to develop an evaluation criteria checklist for 
interactive e-books. Within this perspective, research questions for this study are as follows: 

• What are the core themes to evaluate interactive e-books? 

• What dimensions and criteria should the determined core themes cover? 

Interactive e-Books 

Defining Interactive e-Books 

Books are defined as “the first teaching machine” (McLuhan, 1964, p.174) and they are 
indispensable in the teaching/learning process (West, Turner, & Zhao, 2010). For centuries, 
books have been the catalyst of dissemination and transmission of knowledge. They paved the 
way of improvement, helped to evolve humankind and have evolved themselves. The year 1971 
was a milestone for electronic books (e-books). Michael Stern Hart initiated the Project 
Gutenberg that year to encourage the creation and distribution of e-books (Hart, 2004) and 
created the first digital version of Declaration of Independence as the first e-book in history (Hart, 
1992). Other developments such as the first digital-born hypertext fiction Afternoon in 1980, 
DOS-based e-books and the Runeberg Project in 1992, PDF 1.0 in 1993, E-ink Corporation in 
1997, first handheld e-book reader in 1998, copyright/copyleft and Creative Commons in 2001, 
Kindle e-book reader by Amazon in 2007, and tablet PCs and smartphones at the beginning of the 
new millennium triggered the evolution and acceptance of digital books (Bozkurt, 2013; Bozkurt 
& Bozkaya, 2013a). 
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When comparing definitions, conventional books (c-books) can be defined as a set of written and 
printed sheets that include text and visuals. As a digital version of c-books, Rao (2003) defines e- 
books as text in digital form, a book converted into digital form, digital reading materials, a book 
in a computer file format, an electronic file of words and images displayed on a device screen 
intended for more than solely reading e-books, or an electronic file formatted for display on 
dedicated e-book readers. 

In 2011, introduction of the next-generation digital book required a new definition: interactive e- 
books. In his TED Talk (Technology, Entertainment and Design), Matas (2011) introduced one of 
the first known interactive e-books, Our Choice, and promoted it as a next-generation digital 
book. Our Choice was a clear indicator of the future of digital books as the first full-length digital 
book that utilized various creative and innovative features. Some features of this interactive e- 
book are given in Table 1. 

Table 1 

Next-generation digital book 


• Interaction occurs in multi-channels 

• Interaction is among user, digital book, and environment 

• The book elements interact among themselves 

• Interaction occurs synchronously among many 
components 



QR code for Our Choice 
(TED Talks) 


By examining 20 interactive e-books systematically, Bozkurt & Bozkaya (2013a; 2013b) defined 
interactive e-books as an improved extension of digital books. According to their definition, 
interactive e-books are essentially digital book formats in which the user, the digital book, and 
the environment can interact reciprocally at a high level; digital book elements can 
communicate and interact among themselves and environment as well as users, and many 
communication channels are put in use at one and the same time. They also defined the digital 
book as a generic term that covers e-books, interactive e-books, and other digital book formats. A 
comparison of c-books, e-books, and interactive e-books are provided in Figure 1. 
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Interactive E-Books 



Figure 1. Comparison of c-books, e-books and interactive e-books (Bozkurt & Bozkaya, 2013a; 
2013b). 

According to this definition, it is salient that advanced interactive e-books are at the forefront of 
the digital book evolution which is tightly connected to technological innovation. It can be also 
seen that as a result of e-books’ dependency on technology, the distinction between interactive e- 
books and software and mobile applications is being blurred. However, these blurring borders can 
become distinct by applying design principles of interactive e-books and determining the purpose 
of the application as it refers to the user’s electronic reading (e-reading) experience. In the e- 
reading experience with interactive e-books, there are four type of interactions: interaction 
between environments (real and virtual environments), interaction among the digital book 
elements, interaction with other users, and interaction with the user: 

• Interaction among the digital book elements: Interactive e-book elements can 
communicate and interact among themselves. This refers to the interconnectedness of 
interactive e-book elements (e.g. synchronously retrieving data within the book as in the 
interactive charts). 

• Interaction between environments: Interactive e-books can further communicate 
with real and digital environments. The sensors of an interactive e-book reader 
(accelerometer, barometer, compass, fingerprint reader, gesture sensor, GPS, A-GPS, 
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GLONASS, gyroscope, heart rate monitor, ambient light sensor, proximity sensor and 
more) can gather information from a physical environment, for example, geo-location 
information or data retrieved from online databases. 

• Interaction between digital book and user: In addition to the invisible cognitive 
interaction which occurs while reading in any kind of book, it refers to the tangible 
interaction between an interactive e-book and a user (e.g. detecting user gestures for 
navigation or tracking eyes to keep the screen on in reading mode). 

• Interaction with other users/online communities: Users can interact with a 
specific online community by tracking relevant hashtags related to the content within an 
interactive e-book or they can share all or specific parts of the book on their own online 
social networks. 

Pros and Cons of Interactive e-Books 

Interactive e-books have been preferred increasingly, especially from the beginning of 2000 

onwards for the advantages they provide. Some of the advantages of interactive e-books are listed 

in Table 2. 

Table 2 

Pros of Interactive e-Books 

In terms of users/readers: 

• interactive e-books are portable; you can carry a whole library on one device; 

• they are searchable; readers can find what they need easily; 

• they have an enormous capacity to store a great deal of information in a single book; 

• they have annotating support; readers can edit, add notes, add bookmarks or highlight 
without harming the original work; 

• they are customizable; users can tweak the style according to their needs; 

• they are durable; they do not have a short shelf-life span; 

• virtual libraries can be created; they are always available with a mobile device or through 
cloud computing; 

• if they do not have Digital Rights Management (DRM) restrictions, they are shareable; 

• they make reading accessible for the individuals with special needs; 

• they are cheaper than printed books; they are printable; they can be used like a c-book (if 
it is allowed); 

• they are convertible; they can be used in different formats; 

• they can be hyper-linked for additional sources; 

• they are easy to use with natural user interface and gesture based computing; 

• they are easy to read; they can be read in the darkness; they can be read aloud 
automatically with text to speech; 

• they are portable; there are no weight or bulk constraints; 
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• they defeat attempts of censorship; 

• they support multimedia content and enhance e-reading experiences; 

• they are reusable infinitely, and they promote e-reading. 

In terms of authors: 

• interactive e-books are easy to publish; 

• they empower self-publishing; 

• authors can have feedback from readers and update their book instantly. 

In terms of libraries and other educational institutions: 

• interactive e-books reduce maintenance cost; 

• allow user statistics if featured; 

• can be protected with DRM options; 

• save physical space; 

• a single book can be used by many; 

• readers can take service from libraries 7/24; 

• infinite circulation is possible as interactive e-books do not wear out. 

In terms of publishers and retailers: 

• interactive e-books are environment friendly; 

• they can be delivered almost instantly; 

• they are convenient, they take up no physical space; 

• their publishing speed is faster and publishing cost is cheaper than c-books. 

As well as having advantages, interactive e-books have some disadvantages. However, the 
following disadvantages in Table 3 mostly derive from external reasons, not from the inherent 
features of interactive e-books. 

Table 3 

Cons of Interactive e-Books 

In terms of reading device: 

• resolution of the screens is a problem and longer reading time fatigue eyes; on the other 
hand, new display technologies promise real-like high quality display experiences; 

• compatibility of interactive e-books is another problem; there has yet to be a widely 
accepted universal format that allows all books to be read on any device; 

• reading devices need power so they cannot provide continuous e-reading experience and 
are limited to battery life. 

In terms of piracy: 

• they can be hacked and can be used easily in a manner that violates copyright. 
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In terms of tactile experience: 

• many readers simply enjoy and prefer the smell, weight, and page turning sound of print 
books. 


Literature Review 


Interaction in ODL 

Interaction, as a complex and multifaceted concept in all forms of education, fulfills many critical 
functions in the educational process (Anderson, 2003). It appears to be one of the most critical 
instructional elements (Kearsley, 1995) and it was highlighted that interaction is a necessary 
ingredient for a successful learning experience in ODL (Mclsaac & Gunawardena, 1996). 
According to Dewey (1916), who used the word “transaction” instead of “interaction” to 
emphasize the relationship between organism and environment, interaction is the defining 
component of the educational process that occurs when the learners transform the inert 
information passed to them from another and construct it into knowledge with personal 
application and value. Moore and Kearsley (1996) focused on interaction in distance learning and 
described learner-content, learner-learner, and learner-instructor interaction, while Hillman, 
Willis, and Gunawardena (1994) additionally described learner-interface interaction. Moore and 
Kearsley (1996) further stated that effective teaching at a distance depends on a deep 
understanding of the nature of interaction and how to facilitate interaction through 
technologically transmitted communications. These ideas inspired instructional designers and 
educators not only to design interactivity in learning process, but also to design interactive 
learning tools such as interactive e-books. 

Interaction Design: The Flow between Action and Reaction 

Interaction design (IxD) and Human Computer Interaction (HCI) are terms that have been used 
interchangeably. Currently, there has been a growing interest in the structure and nature of 
interaction design in education and academia. Influenced heavily by ICT in education, interaction 
design became another discipline engaging in education, particularly in online and digital 
learning experiences. 

According to Wagner (1994), within the ODL perspective, interaction is reciprocal events that 
require at least two objects and two actions, and it “occurs when these objects and events 
mutually influence one another” (p. 8). Silver (2007) defines interaction design as a blended 
endeavour of process, methodology, and attitude. Lowgren (2013) states that interaction design is 
about shaping digital things for individuals’ use. 

In essence, interaction design is a system view. All the elements in an interactive system should be 
designed for a purpose (Bozkurt & Bozkaya, 2013a). That’s why interaction design uses five 
dimensions to have a broad view and cover all elements of an interactive system. On this basis, 
the first four dimensions of interaction design were introduced by Smith (2007) and the fifth 
dimension was added by Silver (2007). 
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1 st dimension (Words): Words are interactions that users employ to interact. 


• 2 nd dimension (Visual representations): Visual representations, which include 
typography, diagrams, icons, and other graphics, are the things that the user interacts 
with on the interface. 

• 3 rd dimension (Physical objects or space): It defines the space or objects with which or 
within which users interact. 

• 4 th dimension (Time): Time is the dimension within which users interact. For example, 
content that changes over time such as sound, video, or animation. 

• 5 th dimension (Behavior): Behavior defines the users’ actions, their reactions to the 
interface and how they respond to it. Behavior is about including action, or operation, 
and presentation, or reaction. 

According to Fischer and Coutellier (2005), there are three types of interaction: cognitive, 
sensorial, and pure physical interactions. It is important to define cognitive interaction since any 
type of books, including digital ones, are sources of information that require a cognitive 
interaction to process information and then construct knowledge. Interactive e-books in new 
generation mobile devices cover all these interactions. On this ground, in an interactive e-book 
design process, these three types of interaction should be designed in agreement with planned 
objectives. However, another important issue to decide upon is intended interaction level in 
interactive e-books. 

Interaction Levels 

The distinguishing feature to define and categorize book types (c-book, e-book or interactive e- 
book) is the level of interaction they exhibit. Interactive Multimedia Instruction (IMI), or as it is 
otherwise known, Interactive Courseware (ICW), has been developed over the years by the U.S 
Department of Defense (DoD) (1999). In the ICW model, there are four major levels of 
interactivity which are defined as the degree of student's involvement in the instructional activity. 
The four levels of interactivity identified by the ICW are provided in Table 4. 

Table 4 

Interactivity Levels and Definitions 


Levels 

Description 

Level 1: 

Passive. 

The student acts solely as a receiver of information. 

Level 2: 

Limited participation. 

The student makes simple responses to instructional cues. 
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Level 3: The student makes a variety of responses using varied techniques in 

Complex participation. response to instructional cues. _ 

Level 4: The student is directly involved in a life-like set of complex cues and 

Real-time participation, responses. _ 


Related Research 

The current study attempts to examine a topic which has not been extensively researched as 
interactive e-books are a recent emerging technology. Among the few available studies, Wilson 
and Landoni (2002) prepared an electronic textbook design guideline which had been formed as a 
result of extensive evaluations of electronic books involving around 100 participants from the UK 
higher education. They developed 17 on-screen design guidelines and 5 hardware design 
guidelines. Wilson, Landoni, and Gibb (2002) further discussed the findings emerging from the 
EBONI (Electronic Books ON-screen Interface) Project. Diaz (2003) presented and explained a 
number of evaluation criteria for hypermedia educational e-books to help instructional designers 
and to provide guidance for addressing educational requirements during the design process of an 
e-book. The criteria presented were based on previous research and the author's experiences in 
the development of educational systems and projects. Crestani, Landoni, and Melucci (2006) 
incorporated the results of two separate studies into the design, development, and evaluation of e- 
books: The Visual Book and the Hyper-TextBook projects. The Visual Book project focused on the 
visual component of the book metaphor and The Hyper-TextBook concentrated on the 
importance of models and techniques for the automatic production of functional electronic 
versions of textbooks. The findings gathered in this study demonstrate similarities and confirm 
Wilson and Landoni’s (2002), Diaz’s (2003), and Crestani’s (2006) findings about evaluation 
criteria of e-books. 

Previous research has generally focused on e-books which are usually in a single file format and 
include basic multimedia elements with low interaction. Therefore, it can be argued that there is a 
need to develop evaluation criteria for interactive e-books which include a combination of many 
multimedia components with high interactivity, and the current study intends to fill this gap. 


Theoretical Framework 

Throughout this research, four theories were used to establish a sound base to start and provide a 
clear lens through which we can look, enhance our interpretation, generate valid ideas, and give 
meaning to the research findings. The theories applied in this research were the theories of 
Independent Study, Transactional Distance, Multimedia Richness, and Multimedia Learning. 

Theory of Independent Study: According to Wedemeyer (1981), the essence of distance 
education is the independence of the learners. Wedemeyer’s Independent Study Theory 
emphasizes learner independence and adoption of technology as a way to implement that 
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independence (Simonson et al., 2003). According to the theory, learning can occur in spite of the 
time-space barriers and learning should be individualized by providing wider choices to learners; 
learning responsibility belongs to learners themselves and they learn at their own paces. 

Theory of Transactional Distance: This theory is originally an extension of the Theory of 
Independent Study. Inspired by Dewey’s term of transaction, the Transactional Distance Theory 
refers to the cognitive space between instructors and learners in an educational setting. Moore 
(2007) argues that transactional distance is a typology of all education programs having this 
distinguishing characteristic of separation of teacher and learner. According to Moore (1993), 
transactional distance is a psychological and communication space to be crossed, a space of 
potential misunderstanding between the inputs of instructor and those of the learner. The key 
concepts of the theory are dialogue, structure, and learner autonomy. 

Theory of Media Richness Developed originally by Daft and Lengel (1984), the Media 
Richness Theory is based on the contingency theory and the information processing theory. 
According to the theory, “the more equivocal a message, the more clues and data are needed to 
understand it, and media richness theory places communication mediums on a continuous scale 
that represents the richness of a medium and its ability to adequately communicate a complex 
message” (Carlson & Zmud, 1999, p. 155). The main idea of Media Richness Theory was expressed 
as "the more learning that can be pumped through a medium, the richer the medium” (Lengel & 
Daft, 1988, p. 226). According to the theory, the richness of the media is influenced by four 
criteria (Daft & Lengel, 1984): (1) Capacity for immediate feedback; (2) capacity to transmit 
multiple cues; (3) language variety; and (4) the capacity of the medium to have a personal focus. 

Theory of Multimedia Learning: Proposed by Richard Mayer, this theory explains learning 
with multimedia from the perspectives of educational psychology and e-learning (Ataizi and 
Bozkurt, 2014) and claims that individuals learn better when multimedia messages are designed 
in ways that are consistent with how the human mind works (Clark & Mayer, 2011; Mayer, 2002). 
The theory has three main assumptions (Mayer, 2002): (1) there are two separate channels for 
processing information: the auditory and visual channels (Dual Coding Theory); (2) each channel 
has a limited capacity (cognitive load); and (3) learning is an active process of filtering, selecting, 
organizing, and integrating information through association with previous experiences. 

Methodology 


Research Design 

The study was designed as an embedded mixed model research to provide a better understanding 
of the research problem. The purpose of the embedded design is to collect quantitative and 
qualitative data simultaneously or sequentially. In the embedded design, a secondary form of data 
is used to augment or provide additional sources of information not provided by the primary 
source of data (Creswell, 2004). Throughout this research, the Delphi technique (primary source 
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of data) and heuristic inquiry method (secondary source of data) were applied to obtain and 
analyze data. 

Because the topic was an emerging one dealing with different expertise areas such as instructional 
design and interaction design, and the number of the studies in the literature was insufficient, the 
Delphi technique was preferred as a primary source of data to be able to get expert opinions from 
different disciplines. The Delphi Method is based on a structured process for collecting and 
distilling knowledge from a group of experts by means of a series of questionnaires interspersed 
with controlled opinion feedback (Adler & Ziglio, 1996; Dalkey & Helmer, 1963; Koqdar & Aydin, 
2013). As a highly flexible problem-solving process, it fits for situations where evidence based 
practice is dependent on expert opinion (Sandrey & Bulger, 2006). Expert panel members 
provide feedback, revise judgments, and contribute to the development of agreed-upon practices - 
all with complete anonymity (Flippo, 1998). Within this perspective, the basic assumption of the 
Delphi Method is that the informed, collective judgment of a group of experts is more accurate 
and reliable than individual judgment (Clayton, 1997; Ziglio, 1996). The key characteristics of the 
Delphi technique are defined as anonymity of respondents, controlled feedback process, and 
statistical response (Fowles, 1978). 

User experience (UX) is the essence of the interaction design. Thus, in addition to theoretical and 
practical experiences of Delphi panelists, researchers employed heuristic inquiry to harness 
additional research findings which can be explored by directly engaging research questions. On 
this ground, heuristic inquiry was preferred as a secondary source of data. Heuristic inquiry is an 
experience-based technique for problem solving, learning, and discovery. Douglass and 
Moustakas (1985) define heuristic inquiry as a search for the discovery of meaning and essence in 
significant human experience. The heuristic inquiry is an adaptation of phenomenological 
inquiry, yet it requires the involvement of the researcher in a disciplined pursuit of research 
process (Hiles, 2001; Djuraskovic & Arthur, 2010). 

Sampling 

The selection of panel members is considered to be critical for the Delphi process, which is 
directly related to the focus or objectives of the research (Sandrey & Bulger, 2006). Interactive e- 
books are a final product of different procedures and different expertise. Therefore, it is important 
to select experts for a purpose to apply their knowledge to a certain problem on the basis of 
criteria, which are developed from the nature of the problem under investigation (Hasson, 
Keeney, & McKenna, 2000). 

For this research, the participants are required to be experts in one of the following areas: digital 
books, digital publishing, content design, instructional design, interface and layout design, or e- 
learning. It is further required that they have a background in academic research or experience 
working in the field. Through literature review and snowball sampling, 55 experts were invited to 
the research. A total of 30 experts expressed their intention of participating. A highly 
representative Delphi panel of 30 experts, from well-respected institutions and companies, who 
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have published research and/or had practical and theoretical experience, was constructed to 
contribute to the validity and reliability of research findings. 

In heuristic inquiry, as the secondary form of data, 20 distinguishing interactive e-books were 
selected. A set of criteria were defined to be able to examine representative samples that had 
interactive features peculiar to interactive e-books. The interactive e-books included in the 
heuristic research were those that were most downloaded, awarded, and had positive reviews by 
the critics. Book samples that exhibited different features in terms of the interaction level and 
genre were selected to have maximum variation sampling (Appendix A). 

Procedure 

Before initiating Delphi rounds, a pilot study was carried out with three doctoral students. Delphi 
rounds were arranged based on their feedback. Between December 2012 and March 2013, a total 
of four online Delphi rounds were conducted. To be able to classify ideas hierarchically and 
navigate easily among the emerging criteria, terms are used as theme in macro level, dimension in 
meso level and criteria in micro level throughout the article. 

• First round: In the first round, to be able to elicit creative, innovative, and authentic 
ideas, experts were asked three semi-structured open ended questions. The first and 
second questions were constructed based on the literature review. The first and second 
questions addressed the dimensions and criteria of the content and interface theme. In 
the third question, other possible themes, dimensions, and criteria were asked. The first 
round was completed in three weeks with the participation of all experts of the Delphi 
panel. 

• Second Round: Answers that were provided in the first Delphi round were categorized 
and coded. The findings were organized as themes, dimensions, and criteria. The findings 
were then converted into short simple sentences. As result of the thematic analysis, four 
themes, 33 dimensions, and 98 criteria were organized and listed according to the 
relevant themes. After each item, a comment space was provided. Delphi panel experts 
were asked to evaluate these findings with a holistic perspective, state their ideas for each 
item, and propose new items if they considered it necessary. The second round was 
completed in three weeks with the participation of 29 experts of the Delphi Panel. 

• Third round: Based on the comments and recommendations received in the second 
round, four themes, 19 dimensions, and 109 criteria were identified. In the third round, 
these themes, dimensions, and criteria were presented to experts of the Delphi panel with 
a five-point Likert item. Experts were requested to select “not important, slightly 
important, moderately important, important, and very important” options on a Likert 
item considering the importance and necessity of the each item provided. At the end of 
the questionnaire, experts were asked if they had additional comments regarding the 
third round and questionnaire items. At the end of the third round, items with a mean 
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below 80 percent, median of 4, and interquartile range (IQR) of over 1 were discarded. 
The third round was completed in two weeks with the participation of all experts of the 
Delphi panel. 

• Fourth round: In the final Delphi round, experts of the Delphi panel were requested to 
reevaluate four themes, 19 dimensions, and 49 criteria that met the predefined agreement 
levels with five-point Likert items that emerged in the third round. Percentage, median, 
and IQR statistics used in the third round for consensus were also provided for each item 
to give statistical feedback. At the end of the final round, items with a mean below 90 
percent, median of below 4, and IQR of over 1 were discarded again. As a result of the 
Delphi study, four themes, 14 dimensions, and 33 criteria were obtained. The final round 
was completed in two weeks with the participation of all 30 experts of the Delphi panel. 

In the heuristic inquiry, a total of 20 interactive e-books were downloaded into a tablet computer. 
The interactive e-books were selected purposefully from distinguished examples, most of which 
have notable awards and reviews (Appendix A). Each book was used and examined in four themes 
at the end of the Delphi study. First of all, the validity of the Delphi findings were checked by 
examining the applicability of the criteria which emerged in the Delphi rounds, using 20 
interactive e-books. Following that, researchers systematically noted their experiences as well as 
the features observed. This process was conducted between March and May 2013. All the data 
gathered were coded, categorized, and put into themes using content analysis. The findings that 
matched with the findings gathered in the Delphi study were eliminated to assure that the same 
research findings were not replicated. It is salient that the four new criteria emerged from 
heuristic inquiry are mostly related with user experience which requires direct interaction with 
the products that are investigated. For that reason, it is believed that these four criteria didn’t 
emerge during the Delphi rounds. Lastly, four new criteria were defined and associated with 
relevant themes and dimensions. At the end of the Delphi and heuristic research processes, a total 
of four themes, 15 dimensions, and 37 criteria were identified. 

Data Collection and Analysis 

In Delphi studies, qualitative data can be analyzed using content analysis techniques (Hasson, 
Keeney, & McKenna, 2000). The data in the first and second Delphi rounds of this study were 
analyzed using content analysis. The data were coded, categorized, and put into themes (see 
sample in Appendix B). The findings obtained were converted into short, simple sentences and 
presented as questionnaire items in the third and fourth Delphi rounds. 

For the quantitative data in the third and fourth rounds, statistical methods were used. Consensus 
in a Delphi study is subject to interpretation. Consensus can be decided if a certain percentage 
falls within a prescribed range (Miller, 2006). Some researchers suggest that a level of at least 70 
percent agreement is enough to call a consensus; on the other hand, there is no certain percentage 
defined and it changes according to the scope of the research topic. Other statistics used in Delphi 
studies are measures of central tendency (means, median, and mode) and level of dispersion 
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(standard deviation and interquartile range) (Hasson, Keeney, & McKenna, 2000). To be able to 
get robust and reliable research results, a combination of different statistics were defined as the 
consensus level for the research. Percentage (80% for the third round and 90% for the fourth 
round), IQR, and median were used as statistical indicators of consensus (Table 5). 

Table 5. 

Measurement of consensus 


3rd round Median > 4, IQR < 1, Frequency 4-5 > 80% 

4th round _ Median > 4, IQR < 1, Frequency 4-5 > 90% 


In the heuristic inquiry, both qualitative and quantitative data were gathered systematically 
through real life experiences. The data were then analyzed using content analysis. The findings 
were used to support and to check the Delphi study findings and to discover new findings 
concerning the evaluation criteria of interactive e-books. 

Limitations 

The findings of this research is limited to current Web and mobile technologies in addition to 
capacities of mobile devices used in the research. For this reason, new themes, dimensions, and 
criteria can be added in time with the emergence of new technologies or new learning approaches. 

Reliability 

The data in the first and second round of Delphi were coded by one of the authors (Rater A) of 
this study. Another author (Rater B) who was experienced in the field also rated the data of first 
and second rounds. Cohen’s Kappa (k) was calculated to check the inter-rater reliability of the 
first two Delphi rounds. Inter-rater reliability between Raters A and B for the first round was k = 
.918 (95% Cl, .8551 to .9817), pc.0005 and k = .951 (95% Cl, .9040 to .9981), pc.0005 for the 
second round. Altman (1991) proposed that the extent of agreement can be qualified as poor (c 
0.20), fair (0.21 to 0.40), moderate (0.41 to 0.60), good (0.61 to 0.80), and very good (0.81 to 
1.00). Thus, the reliability of ratings for the first and second round Delphi data can be considered 
as very good. 


Results 

The overall findings of this study are presented in Table 6. In the following table, research 
findings were organized as themes, dimensions and criteria. The criteria were associated with the 
most relevant dimensions and themes, but these criteria may intersect and overlap with other 
dimensions or themes. 

Table 6 
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Interactive e-Book Evaluation Criteria 


CONTENT 

Presentation 

• Clear and fluent language usage 

• Effective narration features 

• Preparing content with a theoretical framework that supports learning objectives 

Richness 

• Richness of multimedia components 

• Balance of information density 

Motivation and Attractiveness 

• Attractiveness of the content 

• Content design appropriate for characteristics of the target audience 

Assessment and Evaluation 

• Providing mechanisms to users enabling them to assess their own learning process 
Integrity , Coherence and Connectivity 

• Content-media coherence 

• Integrity of hierarchical structure of the content 

• Coherence of the content with learning objectives 

• Coherence of language and content 

• Coherence of learning activities with learning objectives 

• Connectivity of content in itself 

INTERFACE 

Ease of Interface Use 

• User friendly interface 

• Navigation tools to avoid disorientation and provide quick access to relevant pages 

• Bookmarking ability 

• Search ability 
Customization and Autonomy 

• Authoring support (e.g. highlighting, note taking, marking etc.) 

Interface Design, Esthetic and Consistency 

• Simple and straightforward layout design 

• Legibility without fatiguing eyes 

• Balanced design of interface density 

• Consistency of buttons, icons and menu items 

• Consistency of interface areas 

• Consistency of labels and messages 

• Providing color balance for layout 
Universal Design for Accessibility 

• Designing under the frame of universal design principles to ease accessibility for 
individuals with special needs (e.g. voice over, scaling visuals or text etc.) 

Support Services 
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• Update support for interactive e-book and content 

Layout Frame Design 

• Applying physical techniques in layout frame design (e.g. balance, symmetry, 
regularity, alignment, proportion and horizontality) 

INTERACTIVITY 

Interaction Richness 

• Designing interactive e-book with components/elements that can provide higher 
interaction level 

Digital book, environment and content interaction 

• Providing interaction through hardware and sensors of the interactive e-book reader 
device* 

• Data feed through the Internet* 

• Supporting interaction and communication through social network connections* 

TECHNOLOGY 

Technical features 

• Online /offline functionality 

• Multiplatform support 

• High resolution and quality of components used in interactive e-book 

Copyright 

• Protecting interactive e-book through DRM* _ 

interactive e-book evaluation criteria which were added after heuristic inquiry 


Discussion 

McLuhan’s (1964) widely known phrase, “The medium is the message,” means that the content of 
a mediated message is secondary to the medium (West et ah, 2010). McLuhan did not ignore the 
importance of the content, but he deliberately pointed out the ability of the medium that shapes 
the message. In this study, the medium’s ability, that is to say interactive e-books, to shape the 
message became apparent in four basic themes that emerged as a result of the data analysis: 
content, interface, interaction, and technology. 

• Content: This theme presents the pedagogic perspective of the evaluation criteria. The 
criteria in this theme highlight the importance of learning/instructional design. It is clear 
that interaction is not provided with solid technologies but with design of the content. 
From the selection of action verbs in the content to the theoretical framework applied, all 
are related with content theme. 

• Interface: This theme is the intersection point where user and interactive e-book contact. 
The theme of interface can be interpreted as the showcase or the face of the interactive e- 
book. The criteria in this theme consider aesthetic and visual design properties. The art of 
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the designer appears in this theme and the criteria are related with usability features of 
the interactive e-book. 


• Interaction: This theme determines the interactivity level of the digital book by using 
interaction design. The questions of interactivity such as what, when, where, how, why, 
and who are answered in this theme. However, it should be noted that interaction is not 
provided merely by using technology, but it is also provided by presentation of the 
content, design of the interface, and use of hardware capabilities of the e-book reader 
device. 

• Technology: This is a challenging theme in contrast to the previous soft themes of 
evaluation criteria. Hardware features and their functions are related with this theme. On 
the other hand, in this study usually hardware features and functions rather than 
hardware were the locus of the study in interactive e-books, which is basically a software 
technology. 

Comparing the results of the current study with Wilson and Landoni (2002) and Diaz (2003), in 
addition to content and interface themes previously defined, two additional themes appeared: 
interaction and technology. E-books which were examined by Wilson and Landoni (2002) and 
Diaz (2003) used mobile devices and dedicated e-book readers as a display tool. However, new 
devices such as smartphones, tablet computers, and new generation dedicated e-book readers 
promise adaptive interaction which employs hard and soft technologies for an immersive user 
experience. These two themes emerged because interaction and technology are a result of fast 
developments in soft and especially hard technologies. Interactive e-books use hardware in an 
innovative way by employing their native features to increase depth of interaction both physically 
and cognitively. The increasing capacities of hard and soft technologies projected themselves as 
new themes in the digital book field. In other words, interactive e-books promise more than 
reading experience; they promise e-reading experience which includes cognitive, sensorial, and 
physical interactions. 

It is also seen that the interactive e-book criteria that emerged in this research are coherent with 
basic principles of four theories which indicate interactive e-books’ value for ODL: 

• Theory of Independent Study: From correspondence study to computer-based distance 
learning, books have been the primary learning material, whether they are conventional 
or electronic. The purpose of using distance learning materials is to provide autonomy to 
learners so that they can learn independently. By providing a range of choices to 
individualize the act of learning, interactive e-books stand as an effective option for 
distance learners. 

• Theory of Transactional Distance: Interactive e-books serve in increasing the level of 
interaction to eliminate physical and psychological barriers between learners and 
learning process and also provide individualized learning experiences by utilizing 
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educational technology. The premise of interactive e-books lies in their ability to act like a 
learning environment more than a knowledge source in which meaningful learning can 
occur. 

• Theory of Media Richness: The essence of the interactive e-books lies in the interaction 
provided. Interactive e-books use various media to increase and support interaction of 
the content using hard and soft technologies. The Media Richness Theory offers a solid 
walkthrough in order to design interaction and communication to augment learning 
through interactive e-books. 

• Theory of Multimedia Learning: Meaningful learning through multimedia requires 
instructional design. Principles of Multimedia Learning Theory provides a proven 
guideline for designers of multimedia learning and helps to improve design of an 
interactive e-book to facilitate learning. 


Conclusion and Future Directions 

In this study, through a mixed research design in which Delphi technique and heuristic inquiry 
were used to gather data, four themes, 15 dimensions, and 37 criteria were developed to be able to 
evaluate interactive e-books. As a significant learning material, this study revealed important 
aspects of interactive e-books to ease access to information and to increase interaction, which are 
needed for meaningful learning experiences for distant learners. 

In this study, a set of criteria was proposed to evaluate interactive e-books. Interactive e-books as 
a learning material encompasses many facets such as interaction design, instructional design, and 
interface design. Furthermore, hardware and software capabilities are directly related with the 
proposed evaluation criteria. Thus, the criteria revealed in this research can be developed 
according to changing needs of learners, and new hard and soft technologies. It is believed that 
the findings gathered in this research will: 

• help develop and evaluate interactive e-books, 

• help instruction and learning designers, software developers, and content providers by 
means of themes, dimensions and criteria identified in this research, 

• be a guide to identify the strengths and weaknesses of an interactive e-book, and 

• be a base for future research. 

As a result of the Delphi study, heuristic design and literature review, the following implications 
can be taken into consideration for future research: 


75 



• Interactive e-books use multimedia elements which provide rich communication 
opportunities. However, as a learning material, there is a need to identify the principles of 
multimedia use and design in interactive e-books to enhance learning experiences and to 
provide effective, efficient, and engaging learning opportunities. 

• Interactive e-books use different communication channels and new technologies to reach 
users. Individuals with special needs (visual impaired, hearing impaired and 
orthopedically handicapped) gain advantages with rich communication channels, gesture 
based interaction, and natural user interface of new generation devices. Thus, it is advised 
to implement future research using universal design principles to meet needs of 
individuals with special needs. 

• Interactive e-books are technology-oriented learning materials. As a source of learning 
and practicing, semantic and adaptive systems can be integrated to increase 
independency and autonomy of learners. Therefore, future research can focus on 
semantic and adaptive systems in interactive e-books to help users track their progress, 
receive feedback, and better assess and evaluate the level of learning. 

As a final remark, this study does not claim that interactive e-books are superior to c-books or e- 
books; conversely, it claims that interactive e-books are a good and flexible alternative to be able 
to provide individualized learning opportunities. On this basis, the themes, dimensions, and 
criteria can be used as a checklist to identify strengths and weaknesses of interactive e-books and 
can be helpful to guide researchers who are interested in interactive e-books as a learning 
material. It is believed that well designed interactive e-books can significantly contribute to 
learning process in an effective and efficient way. 

Acknowledgments 

This study was supported by Anadolu University Scientific Research Projects Commission under 
the grant no: 1303E040. 

Authors' Note 

This study is the reviewed and improved version of the thesis “Defining Evaluation Criteria for 
Interactive E-Books for Open and Distance Learning”, completed in the Distance Education 
Department of Social Sciences Institute at Anadolu University in 2013. 


References 

Adler, M., & Ziglio, E. (1996). Gazing into the oracle. Jessica Kingsley Publishers: Bristol, PA. 
Altman, D. G. (1991). Practical Statistics for Medical Research. London: Chapman & Hall. 


76 



Anderson, T. (2003). Modes of interaction in distance education: Recent developments and 
research questions. In M. Moore (Ed.), Handbook of Distance Education (p. 129-144). 
Mahwah, NJ.: Erlbaum. 

Ataizi, M. & Bozkurt, A. (2014). Book Review: E-Learning and the Science of Instruction. 
Contemporary Educational Technology, 5(2), 175-178. 

Beldarrain, Y. (2006). Distance education trends: Integrating new technologies to foster student 
interaction and collaboration. Distance education, 27(2), 139-153. 

Bozkurt, A., & Bozkaya, M. (2013a). Etkilesimli e-kitap Degerlendirme Kriterleri. Eski§ehir: 
Anadolu Universitesi Yayinlari. Retrieved from 

http://www.academia.edu/60Q70Q7/Etkilesimli e-kitap Degerlendirme Kriterleri 

Bozkurt, A., & Bozkaya, M. (2013b). Etkilesimli E-Kitap: Diinii, Bugiinii ve Yarini. Akademik 
Bili§im 2013. (s.375-381). Akdeniz Universitesi, 23-25 Ocak, Antalya. Retrieved from 
http://www.academia.edu/2f;36Q03/Etkilesimli E-Kitap Dunu Bugunu ve Yarini 

Bozkurt, A. (2013). Aijik ve Uzaktan Ogrenmeye Yonelik Etkilesimli E-kitap Degerlendirme 
Kriterlerinin Belirlenmesi. Anadolu Universitesi, Sosyal Bilimler Enstitiisii, Uzaktan 
Egitim Anabilim Dali. Yiiksek Lisans Tezi. Eski§ehir. Retrieved from 
http://www.academia.edu/2802Q74/Acik ve Uzaktan Qgrenmeve Yonelik Etkilesimli 

E-kitap Degerlendirme Kriterlerinin Belirlenmesi 

Carlson, J., & Zmud, R. (1999). Channel expansion theory and the experiential nature of media 
richness perceptions. Academy of Management Journal, 42(2), 153-170. 

Clark, R. C., & Mayer, R. E. (2011). E-learning and the science of instruction: Proven guidelines 
for consumers and designers of multimedia learning. John Wiley & Sons. 

Clayton, M. J. (1997). Delphi: a technique to harness expert opinion for critical decision-making 
tasks in education. Educational Psychology, 17(4), 373-386. 

Crestani, F., Landoni, M., & Melucci, M. (2006). Appearance and functionality of electronic 
books. International Journal on Digital Libraries, 6(2), 192-209. 

Creswell, J. W. (2004). Educational Research: Planning, Conducting, and Evaluating 
Quantitative and Qualitative Research. Pearson. 

Daft, R.L. & Lengel, R.H. (1984). Information richness: a new approach to managerial behavior 
and organizational design. In L. L. Cummings & B. M. Staw (Eds.), Research in 
organizational behavior (p. 191-233). Homewood, IL: JAI Press. 


77 






Dalkey, N., & Helmer, O. (1963). Delphi technique: characteristics and sequence model to the use 
of experts. Manag Sci, 9, 458-67. 

Dewey, J. (1916). Democracy and education. New York: Macmillan. 

Diaz, P. (2003). Usability of hypermedia educational e-books. D-Lib magazine, 9(3). Retrieved 
from http://www.dlib.org/dlib/march03/diaz/03diaz.html 

Djuraskovic, I., & Arthur, N. (2010). Heuristic Inquiry: A Personal Journey of Acculturation and 
Identity Reconstruction. Qualitative Report, 15(6), 1569-1593. 

DoD, (1999). Department of Defense Handbook: Development of Interactive Multimedia 

Instruction (IMI). Retrieved from http://www.au.af.mil/au/awc/awcgate/dod/hbk3.pdf 

Douglass, B. G., & Moustakas, C. (1985). Heuristic Inquiry: The Internal Search to Know. Journal 
of Humanistic Psychology, 25(3), 39-55. 

Fischer, X., & Coutellier, D. (2006). Research in Interactive Design: Proceedings of Virtual 
Concept 2005. Springer. 

Flippo, R. F. (1998). Points of agreement: A display of professional unity in our field. The Reading 
Teacher, 30-40. 

Fowles, J., (1978). Handbook of futures research. Connecticut: Greenwood Press. 

Hart, M. (1992). The History and Philosophy of Project Gutenberg by Michael Hart. Project 
Gutenberg. Retrieved from 

http://www.gutenberg.Org/wiki/Gutenberg:The History and Philosophy of Project 

Gutenberg by Michael Hart 

Hart, M. (2004). Gutenberg Mission Statement by Michael Hart. Project Gutenberg. Retrieved 
from 

http://www.gutenberg.Org/wiki/Gutenberg:Proiect Gutenberg Mission Statement bv 

Michael Hart 


Hasson, F., Keeney, S., & McKenna, H. (2000). Research guidelines for the Delphi survey 
technique. Journal of Advanced Nursing, 32(4), 1008-1015. 

Hasson, F., Keeney, S., & McKenna, H. (2000). Research guidelines for the Delphi survey 
technique. Journal of Advanced Nursing, 32(4), 1008-1015. 

Hiles, D. (2001). Heuristic inquiry and transpersonal research. Paper presented to CCPE, October 
2001, London. 


78 








Hillman, D. C., Willis, D. J., & Gunawardena, C. N. (1994). Learner-interface interaction in 
distance education: An extension of contemporary models and strategies for parishioners. 
The American Journal of Distance Education, 8(2), 30-42. 

Kearsley, G. (1995). The nature and value of interaction in distance learning. Paper presented at 
the Distance Education Symposium 3: Interaction, University Park, Pennsylvania State 
University. 

Kogdar, S. & Aydin, C. H., (2013). Agik ve Uzaktan Ogrenme Ara§tirmalarmda Delfi Tekniginin 
Kullammi. Anadolu Universitesi Sosyal Bilimler Dergisi. 13(3), 31-44. 

Lengel, R. H., & Daft, R. L. (1988). The selection of communication media as an executive skill. 
The Academy of Management Executive, 2(3), 225-232. 

Lowgren, J. (2013). Interaction Design - brief intro. In M. Soegaard & R. F. Dam (Eds.), The 
Encyclopedia of Human-Computer Interaction, 2nd Ed. Aarhus, Denmark: The 
Interaction Design Foundation. Retrieved from http: / /www.interaction- 
design.org/encvclopedia/interaction design.html 

Matas, M. (2011). A next-generation digital book. TED: Talks in less than six minutes. Retrieved 
from http://www.ted.com/talks/mike matas.html 

Mayer, R. E. (2002). Multimedia learning. Psychology of Learning and Motivation, 41, 85-139. 

Mclsaac, M. S., & Gunawardena, C. N. (1996). Distance education. In D. H. Jonassen (Ed.), 
Handbook of research for educational communications and technology: A project of the 
Association for Educational Communications and Technology (pp. 403-437). New York: 
Simon & Schuster Macmillan. 

McLuhan, M. (1964). Understanding media. New York: Mentor. 

Miller, L. E. (2006, October). Determining what could/should be: The Delphi technique and its 
application. Paper presented at the meeting of the 2006 annual meeting of the Mid- 
Western Educational Research Association, Columbus, Ohio. 

Moore, M. G. (1993). Theory of transactional distance. In D. Keegan (Ed.), Theoretical Principles 
of Distance Education (pp. 22-29). New York: Routledge 

Moore, M. G. (2007). The theory of transactional distance. In M.Moore (Ed.), Handbook of 
distance education (pp. 66-85). Mahwah, NJ: Erlbaum. 

Moore, M. G., & Kearsley, G. (1996). Distance education: A systems view. Belmont: Wadsworth 
Publishing Company. 


79 





Rao, S. S. (2003). Electronic books: A review and evaluation. Library Hi Tech, 21(1), 85-93. 

Rothman, D. (2006). E-books: Why they matter for distance education and how they could get 
much better. Innovate: Journal of Online Education, 2(6). 

Sandrey, M. A., & Bulger, S. M. (2006). The Delphi Method: An approach for facilitating evidence 
based practice in athletic training. Dissertation Abstracts (General), 1979, 334. 

Silver, K. (2007). What Puts the Design in Interaction Design. UX Matters. Retrieved from 

http://www.uxmatters.com/mt/archives/2007/07/what-puts-the-design-in-interaction- 

design.php 

Simonson, M., Smaldino, S., Albright, M., & Zvacelc, S. (2011). Teaching and Learning at a 
Distance: Foundations of Distance Education (5th Edition). Pearson Education. 

Smith, G. C. (2007). Foreword. In B. Moggridge & B. Atkinson (Eds.), Designing Interactions 
(Vol. 17). Cambridge: MIT Press. 

Wagner, E. D. (1994). In support of a functional definition of interaction. American Journal of 
Distance Education, 8(2), 6-26. 

Wedemeyer, C. (1981). Learning at the backdoor. Madison, WI: University of Wisconsin Press. 

West, R. L., Turner, L. H., & Zhao, G. (2010). Introducing communication theory: Analysis and 
application. New York: McGraw-Hill. 

Wilson, R., & Landoni, M. (2002). EBONI: Electronic textbook design guidelines. JISC. Retrieved 
from http://ebooks.strath.ac.uk/eboni/guidelines/ 

Wilson, R., Landoni, M., & Gibb, F. (2002). Guidelines for designing electronic books. In 

Research and Advanced Technology for Digital Libraries (pp. 47-60). Berlin Heidelberg: 
Springer. 

Ziglio, E. (1996). The Delphi method and its contribution to decision-making. Gazing into the 
oracle: The Delphi method and its application to social policy and public health, 3-33. 


80 





Appendix A 

List of interactive e-books examined in heuristic inquiry 


# 

Name 

Genre 

References 

1 

Our Choice 

Science and 
Technology 

• Winner of the 2011 Apple Design Award • Featured in The New York Times. Time 
Magazine, PCWorld, Huffington Post and Gizmodo 

2 

Star Walk 

Science and 
Technology 

• Winner of Apple Design Award 2010 • Over 6 million users 

3 

Bobo Explores 

Science and 
Technology 

• Apple Design Award 2012 Winner • iPad App of the Week • Winner of the KAPi 

Award 2011 for Best Children's App for Older Kids • CTR Editor's Choice Award for 
Excellence in Design • App Tudes' Best Kids' App of 2011 • Selected among best 
kids’ apps of 2011 by The New York Times and USA Today • Kirkus Star Award • 

Kirkus Book Review 2011 Best Kids App • Digital Storytime Star • App My World Best 
iOS App Award • Apple Basic iPad App Award • Apple Hall of Fame Award • Editor’s 
Choice for Best Apps for Kids • Featured in MacLife. Wired, Fast Co Design, The 
iPhone Mom and The Mac Observer 

4 

Back in Time 

History 

• Winner of the WSA Mobile Award 2012 • Selected by The New York Times for the 
TopIO iPad Apps in 2011 • Selected by the American Photo Magazine as one of the 

10 Best Photo E-Books of 2012 • Featured in The New York Times, The Daily, 

Gizmodo and Appmodo • Topi iPad Book in 39 Countries 

5 

Alice 

Story 

• Featured in Gizmodo.com, BBC.co.uk, MacLife.com, lndependent.com, 
TheHuffingtonPost.com, Mashable.com, The Sunday Times and CNET 

6 

The Magic of 

Reality 

Science 

• WINNER of the Digital BookWorld Publishing Innovation Award for Best Non-Fiction 
App 2012 • WINNER of a Best App Ever Award for Best Coffee Table Book App 2012 

• Highly Commended in 2011 Futurebook Awards Best App category • Featured in 

The Next Web, Boing Boing, Recombu and The Times • No.1 book app in 27 
countries 

7 

Paris: DK 
Eyewitness 

Travel 

Featured in New York Times, Washington Post and The Sun 

8 

Mewing Tales' 

Pedlar Lady 

Story 

• iPad App of the week • Awarded Kirkus Star • Kirkus Reviews • iTunes Rewind 

2010 - Interactive Stories • Featured in BestiPadHacks.com 

9 

March of the 
Dinosaurs 

Science and 
Technology 

• Featured in Know Your Apps and App Advice 

10 

The Wrong Side of 
the Bed in 3D 

Story 

• Selected as one of the 10 Best Children's Book Apps by Kirkus Reviews, iPhoneLife 
&Babble.com • Featured in CNET, The New York Times & Huffington Post 

11 

Sports Illustrated 
Kids 3D 

Sports 


12 

National 

Geographic Birds 

Science and 
Technology 

• A School Library Journal Top App of 2013 

13 

Solar System 

Science and 
Technology 

• Winns’ of the FutureBook Digital Innovation Prize (The Bookseller Industry Awards) 

- March 2011 • FutureBook Digital Innovation Award • featured in WIRED Magazine 
and USA Today Post 

14 

LIFE Digital Photo 
Guide 

Photography 


15 

War in the Pacific 

History 

• 2nd Best selling book on the US App Store 

16 

Ultimate 

Science and 

• WINNER of the Digital BookWorld Publishing Innovation Award for Best Non-Fiction 


Dinopedia: 

Complete 

Dinosaur 

Reference 

Technology 

App 2012 • WINNER of a Best App Ever Award for Best Coffee Table Book App 2012 
• Highly Commended in 2011 Futurebook Awards Best App category • New York 

Times Top 10 Best Children's Books on the iPad • 2011 MOBI Award Winns • 
Time.com Top 25 iPad Apps for Kids • Chosen as New and Noteworthy, What's Hot, 
and Staff Favorites by Apple • Appy Award Finalist • 5 Stars from Appolicious • 

KinderTown 10 Top Apps Kids Will Beg to Play • No. 1 book app in 27 countries 

17 

Dictionary.com 
Dictionary & 
Thesaurus 

Reference 

CNET Best 100 App Award 

18 

PopOutl The Tale 
of Peter Rabbit 

Story 

• Kirkus Top Book App of 2010 • Winner of Parents' Choice Gold Award • iPad Hall of 
Fame • iPad Apps Starter Kit • Children's Tech. Review Editor's Choice • iPhone and 
iPad App of the Week • Common Sense Media - 5 Stars • FWA Mobile of the Day - 
Feb 2011 • CTR Best Apps for Kids 2010 • iTunes Rewind - Top Storytelling App • 
Featured in Mashable, NY Times, The New Yorker, NPR, FOX SLJ 

19 

The Elements 

Science and 
Technology 

• Featured in USA Today and Boing Boing 

20 

Journey to the 

Science and 

• APP OF THE WEEK • The International #1 iPad Book App • Bronze Medal Winner: 


Exoplanets 

Technology 

Lovie Award for Best Tablet App, Education & Reference • FWA Mobile of The Day 
(MOTD) Award • Shortlisted, FUTUREBOOK AWARD • Selected one of USA Today's 
“10 Coolest Book Apps for Fall" 


Appendix B 

Example of content analysis and procedure for coding and categorizing according to themes, 
dimensions and criteria 
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Phase 1: Questions to elicit themes, dimensions and criteria 

Round 1, Question 3. 

Apart from content and interface, what other themes, dimensions and criteria should be used to evaluate 

interactive e-books? 

Phase 2: Content analysis process 

Delphi panelists’ comments 

Defining relevant themes, 

dimensions and criteria 

P27: When approached in terms of interactive e-book 
developers, “technology” should be examined as a theme... 

• Technology (Theme) 

P6:... Technical specs should be included as an evaluation 

measure.... 

0 Technique (Dimension) 

P9: ...The digital book shouldn’t depend on a specific operation 
system or device... Interactive e-book should perform well in 

dedicated e-book readers... 

■ Multiplatform support (Criteria) 

P26: Interactive e-books can be evaluated in terms of 

accessibility such as online/offline functionality or accessibility 
from different devices such as tablet pc and smart phones in 

addition to laptop or desktop computers... 

■ Multiplatform support (Criteria) 

• Online /offline functionality' 

(Criteria) 

P28: Digital books should function in different mobile 

operation systems... 

■ Multiplatform support (Criteria) 

P8: ....the content of the interactive e-books should work on 

different platforms or should provide support for different 
platforms... 

■ Multiplatform support (Criteria) 

P29: Functionality in different types of devices should be one 

of the evaluation criteria. 

■ Multiplatform support (Criteria) 

Phase 3: Classification Schema 

• Technology (Theme) 

0 Technique (Dimension) 

■ Multiplatform support (Criteria) 

• Online /offline functionality' (Criteria) 
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